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I.  Introduction 

A source  emits  symbols  drawn  from  the  alphabet  {l,2,...  c};  symbol 
i has  probedsility  p^.  The  source  symbols  are  encoded  into  d-ary  code- 
words. The  codeword  corresponding  to  symbol  i has  leztgth  m^. 

It  is  well  known  that  Huffman's  procedure  [2]  yields  a prefix  con- 
c 

dition  code  minimizing  ^ P^  m. . We  show  in  Section  II  that  it  can 

i-1  ^ *■ 

be  generalized  to  yield  a prefix  condition  code  minimizing  the  mceient 

c sm. 

generating  function  I P^  e for  a given  s > 0. 

i-1 

A dravft>ack  of  transmitting  veuriable  length  codewords  over  a synchronous 

line  is  the  need  for  a buffer  at  the  output  of  the  encoder.  This  buffer 

may  overflow.  The  probability  that  a source  symbol  causes  an  overflow 

-s*B 

is  of  the  order  of  e , as  can  be  shown  by  Wyner's  results  [6].  In 
this  fonmila  B denotes  the  buffer  size  emd  s*  is  the  largest  s (possibly 
such  that 

c sm. 

A(s)  I P.  e ^ < 1 (1) 

i=l 


where  A(s)  - E[e^(-st)]  denotes  the  Laplace-Stieltjes  tramsform  of  the 


probability  distribution  of  the  source  intermission  times  measured  in  units  of 


encoded  digit  transmission  time.  This  result  requires  the  mutual  independence 


intermission  time  is  greater  than  the  average  codeword  length.  It  also 


ignores  the  need  to  indicate  to  the  receiver  when  the  buffer  is  empty 


(or  overflows) 


It  is  thus  desirable  to  use  a code  with  s*  as  large  as  possible  when 


asymptotic  approximation  is  good.  The  use  of  s*  as  a criterion  was  first 
suggested  by  Jelinek  [4] . The  search  for  such  a code  is  the  subject  of 
Section  III.  We  consider  only  the  problem  of  symbol  by  symbol  encoding. 


w saa. 

II.  Minimization  of  J,  Pj  e s > 0. 

i-1  ^ 

Without  loss  of  essential  generality  we  cam  assume  c = d + k(d-l)  for 
some  integer  k.  (so  that  c is  the  number  of  terminal  nodes  in  ; - 
some  d-ary  trees)  . amd  i ■ 1,2,...  c-1.  It  is  yell  known 

C 8XD  • 

[1,  p.49]  that  no  gain  cam  be  achieved  by  minimizing  ^ P,  e over  all 

i-1  ^ 

uniquely  decodable  codes,  rather  tham  only  over  the  prefix  condition  codes. 

sm.  analogously  to  the  argument 

Because  s > 0,  e increases  with  m^,  thus,(«^  given  by  Huffman  [2] , 

there  is  an  optimail  prefix  condition  code  where  the  codewords  corresponding 

to  symbols  c-d+l  to  c are  the  longest  and  differ  only  in  the  last  character. 

If  c-d  this  observation  specifies  am  optimal  code.  If  c > d this  reduces 

the  problem  to  finding  a prefix  condition  code  of  size  c-d+1  « d -f  (k-1)  (d-1) 

minimizing 


c-d  s 

I Pi  • 
1-1  ^ 


+ (e* 


® sm 

1 p.» 

1-c-d+l 


Thus  the  "merge*  step  in  the  Huffman  construction  is  replaced  here  by 
"merge  amd  scale  by  e*”. 

Again  we  cam  make  the  same  observation  and  continuing  we  will 
eventually  reach  the  point  tdiere  the  code  is  completely  specified. 

For  s close  enough  to  0,  this  algorithm  yields  a Huffmam  code,  since 
sm^  c sm^ 

e ^ 1+sm^,  so  that  minimizing  ^ P^  e is  equivalent  to  minimizing 
c i-1 

^ p . m. . For  s laurge  enough,  this  algorithm  assigns  codewords  of  length 
i-1 
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III.  Finding  A Prefix  Condition  Code  With  Larqtst  s*. 

Following  {4}  we  first  note  that  it  is  possible  to  upperbound  s* 
over  all  uniquely  decodable  codes.  By  Holder's  inequality,  for  s > 0, 


(c  sm.  \ In  d+s  / c -m,  \ In  d+s 

) (J.  * ) 


In  d 
In  d-t-s 


> £ 


1-1 


c — m. 

For  a uniquely  decodable  code,  ^ d < 1 (1,  p.  47];  thus 

i-1 


snt. 


W 0MS  B 

A(s)  2 P,  e ^ > A(s) 

i-1  ^ 


(l 

Vi-I 


In  d 
In  d+s 


) 


In  d+s 
In  d 


(2) 


Consequently  the  s*  corresponding  to  a uniquely  decod<d>le  code  is  not 

greater  tlum  s^,  defined  as  the  largest  s such  that  the  right 

■esiber  of  (2)  is  less  than  or  equal  to  1.  In  general  s^  is  not  achievable 

because  the  m^'s  must  take  integer  values.  The  right  ...  /seraber  of 

(2)  is  a convex  function  of  S;  sb  s-O,  its  value  is  1,  while  the  value 

of  its  derivative  is  equal  to  the  entropy  (base  d)  of  the  source  alphabet 

latter 

minus  the  mean  intermission  time.  Thus  if  thlsj^quantity  is  negative  then 
s^  is  positive,  and  conversely  except  in  the  degenerate  case  where  the 


right 


member  of  (2)  is  Identical  to  1. 


For  a given  code,  C,  we  denote  the  corresponding  A(s)  J p.  e 

i-1 

by  f(C,s).  f(C,s)  is  the  Laplace->Stieltjes  transform  of  the  probability 


dlstrllMtlon  of  an  Intemisslon  tine  minus  a codeviord  length,  thus  it 
is  convex  in  s eu^  f(C,0)  » 1. 


The  iterative  algorithm  to  find  a prefix  condition  code  with  largest 
8*  is  as  follows  (see  also  Figure  1) . 

1.  Choose  any  s^  in  lO,*]  (a  good  choice  is  s^  “ s^) 

2.  j 0 

3.  j j+1 

4.  Use  the  algorithm  of  Section  II  to  find  a code  C.  minimizing  ~ 

c sn^  3 

I -P,  e ^ for  s - 8, 
i-l 


j-1* 


5.  Compute  the  s*  corresponding  to  C^.  Denote  it  by  s^. 

6.  If  s^  3,  else  stop 

Of  course  we  must  show  that  this  algorithm  terminates  and  that 


the  last  code  generated  is  optimal. 

First  we  note  that  s^^^  21  j > 1,  because 

I (line  5)  f(C.,  s.)  < 1 and 

I J J ~ 

(line  4)  s^)  < f(C^»  a^) 

and  the  definition  of  a^^^.  Secondly,  we  observe  that  the  number  of  codes 
that  can  possibily  be  generated  by  the  algorithm  of  Section  II  is  finite, 
if  only  because  all  codeword  lengths  must  be  smaller  than  c.  These  two 
remarks  insure  that  the  algorithm  will  terminate  in  a finite  time. 

Let  C^  ^uld  s^  denote  respectively  the  last  generated  code  and  its 
s*,  while  s denotes  the  leirgest  achievaUble  s*.  We  must  show  that  s^  « s. 


/ 
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For  the  sake  of  brevity  (the  complete  proof  appears  in  [ 3] ) we  assume  here 

that  f(C,s)  is  not  identical  to  1 when  C is  an  optimal  code;  this  happens 

when  the  intermission  times  eure  equal  with  probability  one  to  an  integer  t, 

emd  when  the  number  of  source  symbols  with  non  zero  prob«d>ilities  lies 

between  d^  - d -f  2 amd  d^.  With  this  restriction  f(C,s)  is  strictly 

convex,  f (C,0)  * 1 and  f(C,S)  £ 1 when  C is  an  optimal  code.  If  s^  * <», 

then  is  optimal.  If  s^  < <»,  then  f(C^,s^)  >1.  If  s^  > 0 amd 

s^  < s then  by  strict  convexity  f(C,s^)  < 1 when  C is  em  optimal  code, 

2U)d  Co  may  not  be  the  last  generated  code.  If  s.  » 0 then  ^ f (C-,  s) I „ > 0 
x>  X ds  X ‘ s=o  — 

and,  as  we  have  seen  in  Section  II,  is  a Huffmeui  code  and  therefore 
^ uniquely  ^ 

minimizes  — f (C,s)  over  all^decodedjle  codes.  Thus  — f (C,s)  ^ 0 

for  all  uniquely  decod2d3le  codes,  euid  by  the  strict  convexity  argximent 

s = 0. 

This  algorithm  was  tested  on  a typical  128  symbol  alphabet,  for  Poisson 

and  deterministic  intermission  ' processes,  with  s^  > s^.  Convergence 

wsLS  fast  (1-2  iterations)  in  the  Poisson  case,  slower  (3-10  Iterations) 

^ in  the  deterministic  case.  The  relative  difference  between  s auid  s* 

an  ordinary 

corresponding  to /(  Ruffistan  code  ranges  from  » (deterministic,  light 
traffic)  to  10%  (Poisson,  light  traffic) , to  0 (heavy  traffic) . 

Jelinek  emd  Schneider  [5]  give  an  algorithm  for  constructing  variable 
I length  to  block  codes  which  is  the  dual  of  the  method  given  in  Section  II. 

They  claim  that  when  this  algorithm  is  used  with  s » s^  then  a vari2d}le 
length  to  block  code  with  largest  achievaOsle  s*  results.  It  seems  that 
this  is  not  necessarily  true  and  that  an  iterative  procedure  analogous 
to  the  one  developed  in  Section  III  could  be  applied  to  obtain  an  optimal 
' I code. 
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